Oct 1, 2013 9:19am
"retroactive and permanent"
I have a site that I like to view from time to time and I am concerned, with the current rules, it may not be available for future individuals.http://portal.stodge.org
(Page cannot be crawled or displayed due to robots.txt.)
From your Frequently Asked Questions:
" Currently there is no way to exclude only a portion of a site, or to exclude archiving a site for a particular time period only. When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent. "
Previously, the owner wanted this to be Archived:http://everything2.com/title/Portal%253A+A+Dataspace+Retrieval?author_id=888590#salimfadhley
(This is an online version of a book called "Portal: A Dataspace Retrieval")
"The pages are designed to be search engine friendly – should a crawler bump into one of the pages the entire site should be absorbed in no time at all. I hope people will soon start bumping into theses pages at random, and then perhaps start reading this odd story. "
Am I to understand correctly from your Frequently Asked Questions, that if a site posts a new robot.txt with the correct parameters to not be index, it will be retroactive? Including previous years?
Will this ever change?
Additionally I want to thank you for a great service and I have been amazed with what you have been able to archive.